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1 Introduction 



We have developed a practical algorithm for state-machine replication [7, 1 1] that tolerates Byzan- 
tine faults. The algorithm is described in [4]. It offers a strong safety property — it implements 
a linearizable [5] object such that all operations invoked on the object execute atomically despite 
Byzantine failures and concurrency. Unlike previous algorithms [11, 10, 6], ours works correctly 
in asynchronous systems like the Internet, and it incorporates important optimizations that enable 
it to outperform previous systems by more than an order of magnitude [4]. 

Since Byzantine-fault-tolerant algorithms are rather subtle, it is important to reason about them 
formally. This paper presents a formal specification for the unoptimized version of our algorithm 
presented in Section 4 of [4] and proves its safety (but not its liveness.) The specification uses the 
I/O automaton formalism of Tuttle an Lynch [8] and the proof is based on invariant assertions and 
simulation relations. [ 

The specification and proof presented in this paper have some interesting, novel properties 
that are independent of our algorithm. First, we use an I/O automaton to formalize the correct 
behavior of our Byzantine-fault-tolerant object implementation. This technique has been used for 
benign failures [8] but we believe we are the first to use it for Byzantine faults. The advantage of 
using an I/O automaton to formalize the correct behavior is that it enables the use of state-based 
proof techniques like simulation relations. These techniques are more stylized than trace-based 
proof techniques — they are more convincing and they are amenable to machine verification. 
Second, our formalization accounts for Byzantine faults of both replicas and clients. A trace-based 
formalization of linearizability in the presence of Byzantine-faulty clients [9] has been proposed 
recently. Our formalization has the advantage that it enables the use of simulation relations. And 
it differs from the one in [9] because it makes authentication and access control explicit in the 
formalization. Revocable access control is a powerful defense against Byzantine faulty clients. 
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'The paper assumes the reader is familiar with I/O automata, invariant assertions, and simulation relations. Lynch's 
book [8] provides a good description of the formalism and the two proof techniques. 



Third, we structure our proof such that our assumptions about authenticated communication are 
isolated in a small number of invariants and in the proofs of a small number of simulation steps. 
This leads to a simpler proof. 

The paper is organized as follows. Section 2 presents the high-level model for the system 
and our assumptions. Section 3 formalizes the correctness condition for our algorithm using a 
simple I/O automaton S as a specification of correct behavior. Section 4 defines the automata that 
compose our replicated system implementation. But it does not attempt to explain the algorithm. 
The reader is referred to [4] for a natural language description of the algorithm that should be easier 
to understand. This section also proves the safety of the algorithm by using invariant assertions and 
simulation relations to show that it implements S. The automata defined in Section 4 implement 
a simplified version of the algorithm that does not garbage collect information. Section 5 defines 
a version of the algorithm with garbage collection and proves its safety by using a simulation 
relation to show that it implements the simplified version of the algorithm in 4. 



2 Model 

The goal of our algorithm is to provide a Byzantine-fault- tolerant implementation of an atomic 
object [8] for a given variable of some type T. Our atomic object implementation uses replication 
to enable concurrent sharing of the variable by many clients in a distributed system. It ensures 
linearizability [5] — all operations invoked on the variable execute atomically despite Byzantine 
failures and concurrency. We start by defining the variable type Tand then describe the architecture 
of the atomic object implementation. 

Variables of type T have a value in a set V, which is initially equal to v . Their behavior is 
defined by the function: 

g:CxOxV^O'xV 

The arguments to the function are a client identifier in C, an operation in a set O (which encodes 
an operation identifier and any arguments to that operation) and an initial value. These arguments 
are mapped by g to the result of the operation in O 1 and a new value for the variable. We require 
g to be total. This can be achieved in practice by having g map all pairs with an invalid operation 
to a pair with an error result and the argument value. 

The client identifier is included explicitly as an argument to g to make it clear that g can return 
different results for different clients. In particular, g can perform access control; if the client is not 
allowed to perform the argument operation, g can return a special no-access error and leave the 
variable's value unmodified. Additionally, access control can depend on the state of the service 
thereby allowing atomic access revocations. Access control with revocable access is an important 
defense against Byzantine-faulty clients. 

We model the atomic object implementation and its clients as a set of I/O automata [8]. Each 
client has a unique identifier c in a set C and is modeled by a client automaton C c . The composition 
of all clients is denoted by C. The atomic object automaton A is the composition of three types 
of automata: proxy, multicast channel, and replica. Figure 1 shows the architecture of the system 
and Figure 2 presents the external interface of A. 
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Figure 1 : Implementation Architecture 



There is a proxy automaton P c for each client C c . P c provides an input action for client c to 
invoke an operation o on the shared variable, REQUEST(o) c , and an ouput action for c to learn the 
result r of an operation it requested, REPLY(r) c . The communication between C c and P c does not 
involve any network; they are assumed to execute in the same node in the distributed system. P c 
communicates with a set of server replicas to implement the interface it offers to the client. 

Each replica has a unique identifier i in a set 1Z and is modeled by an automaton Ri. We 
assume \1Z\ = 3/ + 1 for some positive integer /. This threshold / is the maximum number of 
replica faults that can be tolerated by the system. The resiliency of our algorithm is optimal: 3/ + 1 
is the minimum number of replicas that allow an asynchronous replication system to implement 
an atomic object when up to / replicas are faulty (see [3] for a proof.) 

We assume replicas execute in different nodes in the distributed system. Communication 
between a proxy and the set of replicas and among replicas is performed using a multicast channel 
automaton MC. Automata have no access to the state components of automata running on other 
nodes in the distributed system. 

The multicast channel automaton MC may fail to deliver messages, it may delay them, 
duplicate them, or deliver them out of order. We do not assume synchrony. The nodes are part of 
an asynchronous distributed system with no known bounds on message delays or on the time for 
automata to take enabled actions. 

We use a Byzantine failure model, i.e., faulty automata may behave arbitrarily (except for 
the restrictions discussed next.) The CLIENT-FAILURE and REPLICA-FAILURE actions are used to 



model client and replica failures. Once such a failure action occurs the corresponding automaton 
is replaced by an arbitrary automaton with the same external interface and it remains faulty for 
the rest of the execution. We assume however that this arbitrary automaton has a state component 
called faulty that is set to true. It is important to understand that the failure actions and the faulty 
variables are used only to formally model failures for the correctness proof; our algorithm does 
not know whether a client or replica is faulty or not. 

Input: request(o) c , o e 0,c e C 

CLIENT-FAILURE C , C G C 
REPLICA-FAILURE; , i G 11 

Output: REPLY(r) c , r e G',c e C 

Figure 2: External Signature of A 

We allow for a very strong adversary that can coordinate faulty nodes, delay communication, 
or delay correct nodes in order to cause the most damage to the replicated service. But we 
assume two restrictions on the adversary and the faulty nodes it controls: automata can use 
unforgeable digital signatures to authenticate communication; and they can use collision-resistant 
hash functions. These assumptions are defined in more detail next. 

Unforgeable signatures: Any non-faulty client proxy or replica automaton, x, can authenticate 
messages it sends on the multicast channel by signing them. We denote a message m signed by x 
as (m) (Tx . And (with high probability) no automaton other than x can send (m) (Tx (either directly 
or as part of another message) on the multicast channel for any value of m. 

Collision-resistant hash functions Any automaton can compute a digest D(m) of a message m 
such that (with high probability) it is impossible to find two distinct messages m and m' such that 
D(m) / D(m'). 

These assumptions are probabilistic but there exist signature schemes (e.g., [2]) and hash 
functions (e.g, [1]) for which they are believed to hold with very high probability. Therefore, we 
will assume that they hold with probability one in the rest of the paper. 



3 Correctness Condition 

We specify the correct behavior for A by using another I/O automaton S with the same external 
signature as A. We say that A is correct if it implements S. S is a simple abstract atomic object 
for a variable of type T that is defined as follows: 



Signature: 




Input: 




request(o) c 






CLIENT-FAILURE,; 






REPLICA-FAILUREi 


Internal: 




EXECUTE(o,t,c) 






faulty-request(o, t, c) 


Output: 




REPLY(r) c 


Here, o G 


0,\ 


! e N, c e C, i e 11, and r e C' 


State: 







va/ € V, initially u 

m C (!) x N x C, initially {} 

out C £>' x N x C, initially {} 

V c € C, last-req c G N, initially last-req c = 

Vc G C, last-rep-t c G N, initially last-rep-t c = 

Vc G C, faulty -client c G 5ooZ, initially faulty-client c = false 

y i & 1Z, faulty-replica i G Boo/, initially faulty -replica { = false 

n-faulty = \{i\faulty-replica i = true}\ 



Transitions (if n-faulty < /): 

request(o) c 
Eff: last-req c := last-req c + 1 
in := inU {(o, last-req c , c)} 

CLIENT-FAILURE C 
Eff: faulty-client c := frwe 

REPLICA-FAILUREi 
Eff: faulty-replicai := frMe 

REPLY(r) c 
Pre: faulty- client c = trueV 3t : ((?■,*, c) G owf 
Eff: om? := out •^■{(r, t,c)}) 



faulty-request(o, t, c) 
Pre: faulty- client c = true 
Eff: in := in U {(o, t, c}} 

EXECUTE(o,t,c) 

Pre: (o, t, c) G in 

Eff: m := in •^■{(o, t, c}} 
if t > last-rep-t c then 
(r, va/) := g(c, o, va/) 
out := ot(f U { (r, t, c) } 
last-rep-t c := t 



Most of the definition of 5 is self-explanatory but some issues deserve clarification. To 
model the fact that A does not behave correctly when more than / replicas are Byzantine-faulty, 
the behavior of S is left unspecified when n-faulty > /, i.e., S may behave arbitrarily with the 
restriction that the faulty-client and faulty '-replica variables that have value true cannot be modified. 
The FAULTY-REQUEST actions model execution of requests by faulty clients that bypass the external 
signature of A, e.g., by injecting the appropriate messages into the multicast channel. Similarly, 
the REPLY precondition is weaker for faulty clients to allow arbitrary replies for such clients. 

The last-req c component is used to distinguish requests by c to execute the same operation 
o. And, last-rep-t c remembers the value of last-req c that was associated with the last operation 
executed for c. This models a well-formedness condition on non-faulty clients: they are expected 
to wait for the reply to the last requested operation before they issue the next request. Otherwise, 
one of the requests may not even execute and the client may be unable to match the replies with 
the requests. 



4 The System 



This section defines the multicast channel, proxy, and replica automata. 



4.1 The Multicast Channel Automaton 

The multicast channel automaton models the communication network connecting the proxy and 
replica automata. There is a single multicast automaton in the system with SEND and RECEIVE 
actions for each proxy and replica. These actions allow automata to send messages in a universal 
message set M. to any subset of automata with identifiers in X = C U 1Z. The channel automaton 
does not provide authenticated communication; the RECEIVE actions do not identify the sender of 
the message. It is defined as follows. 

Signature: 

Input: sEND(m, X) x 

Internal: misbehave^, X, X') 

Output: RECEIVE(m) a; 

Here, m 6 M, X, X' C X, and x E X 

State: 

wire C M x 2*, initially {} 



Transitions: 

SEND(m, X) x 
Eff: wire := wire U {(m, X)} 



MISBEHAVE(m, X, X') 
Pre: (m, X) € wire 
Eff: wire := wire <=>{(m, X)} U {(m, X')} 



RECEIVE(m) a; 
Pre: 3(m,X) e wire : (x G X 
Eff: wire := wire <=>{(m, X)} U {(m,X -&{x})}) 

The MISBEHAVE actions allow the channel to loose messages or duplicate them and the RECEIVE 
actions are defined such that messages may be reordered. Additionally, the automaton is defined 
such that every message that was ever sent on the channel is remembered and can be replayed 
later. 



4.2 The Proxy Automaton 



Each client C c Ointeracts with the atomic object through a proxy automaton P c , which is defined 
as follows. 



Signature: 

Input: request(o) c 

RECEIVE( (REPLY, V, t, C, i, r} CT; ) c 
CLIENT-FAILURE,; 
Output: REPLY(r) c 

SEND(m, X) c 

Here, o G O, v, t G N, c G C, i G 11, r G O' , m G M, and X C X 

State: 

vieWc G N, initially 
in c C .M, initially {} 
out c C A4, initially {} 
last-req c G N, initially 
retrans c G Bool, initially /a/se 
faulty c G Bool, initially /a/se 

Transitions: 

request(o) c 
Eff: last-req c := last-req c + 1 

owf c := {(request, o,tef-re^ c ,c) CT< .} 
r« c := {} 
retrans c := false 

receive( (reply, d, i, c, «, r) CT; ) c 
Eff: if [out c ^ {} A last-req c = i) then 

!M C := in c U {(REPLY, v, t, c, «, r) CT; } 

CLIENT-FAILURE C 
Eff: faulty c := true 

REPLY(r) c 
Pre: o«f c ^ {} A 3R : (\R\ > f A V« G R : (3v : ((REPLY, v, last-req c ,c, i, r) <Ti G in c ))) 
Eff: view c := max({v\{REPLY,v,last-req c ,c,i,r) c , i £ in c }) 
out c := {} 

SEND(m, {vieWc mod |7£|}) c 
Pre: m G owf c A ->retrans c 
Eff: retrans c := true 

SEND(m, 7^) c 
Pre: m G out c A retrans c 
Eff: none 



4.3 The Replica Automaton 

Each replica automaton E4 is denned as follows. 



Signature: 

Input: RECEIVE ((REQUEST, O, t, c) a<: )i 

receive((pre-prepare, V, n, m) <7j )i 
receive( (prepare, V, n, d, j) ai )i 
receive( (commit, V, n, d, j) aj )i 
receive( (view-change, V, P, j) aj )i 
receive( (new- view, v, V, O, N) aj ); 

REPLICA-FAILURE; 

Internal: send-pre-prepare(77i, v, n), 

SEND-COMMIT(m, V, n), 
EXECUTE(m,v,n), 
VIEW-CHANGE(u)i 
SEND-NEW-VIEW(i;, V), 
Output: SEND(m, X) c 

Here, t, v,n G N, c G C, i, j G 11, m G M, V, 0, N C M, X C X, and 
d G V = {d | 3m G M : (d = D(m))} 

State: 

vali G V, initially v 

viewi G N, initially 

irii C jM, initially {} 

owfi C A4, initially {} 

last-rep i : C —¥ O' , initially V c G C: last-rep ; (c) = null-rep 

last-rep-t i : C — > N, initially V c G C : last-rep-t i (c) = 

seqno i G N, initially 

last-execi G N, initially 

faulty i G Bool, initially /a/se 

Auxiliary functions: 

tag(m,u) = m=(u,...) 

primaryiv) = v mod | 7?.| 

primary(i) = viewi mod | 7?.| 

in-v(v, i) = viewi = v 

prepared(m,v,n,M) = {PRE-PREPARE,v,n,m) <Tpr i mar . (v) £ MA 

3R : (\R\ > 2/ A primary(v) £ R A V k € R : ((PREPARE, v, n, D(m), k) ah G M)) 
prepared(m,v,n,i) = prepared(m,v,n,irii) 
last-prepared(m, v, n, M) = prepared(m, v, n, M) A 

3m',v' : ((prepared(m' ,v' ,n,M) A v' > v) V (prepared{m! ,v,n, M) A m ^ m!)) 
last-prepared(m,v,n,i) = last-prepared(m,v,n,irii) 
committed(m,v,n,i) = (3v' : ((PRE-PREPARE, v',n, m) a primary r„n e !W *) V m G in;) A 

3fl : (|E| >2/ + lAVA:G.R: ((COMMIT, «, n, £>(m), fc)^ G an)) 
correct-view-change(m,v,j) = 3P: (m = (VIEW-CHANGE, v, P, j) a . A 
V (PRE-PREPARE, v' ,n, m!) a „ r i ma ry( , G P : (last-prepared(m' ,v' ,n,P) A v' < v) 
merge-P(V) = { m \ 3 (VIEW-CHANGE, v, P, k) ak G V : m G P } 
max-n(M) = max({ n | (PRE-PREPARE, v, n, m) cri G M}) 
correct-new-view(m, v) = 
3V,0,N,R: (m = (new-view, v, V, O, N)^ primaryM A \V\ = \R\ = 2/ + 1 A 
V k G R : (3 m' G V : (correct-view-change(m' , v, k))) A 

O = { (PRE-PREPARE, v,n, m ')<Tr, r i ma ry(v ) I 3 v' : last-prepared(m' ,v' ,n,merge-P(V))} A 
N = { (pre-prepare, v, n, null) a primary^) I n < tnax-n(0) A 



3 v' ,m! ,n : last-prepared(m' ,v' ,n, merge-PiV))) 
has-new-view(v,i) e » = V 3m : (m € in; A correct-new-view(m, v)) 

Input Transitions: 

receive( (request, O, t, c) CTc )j 
Eff: let m = (REQUEST, o, t, c) tTc 
if t = last-rep-t^c) then 

outi := ouu U {(reply, vieWi,t, c, i, last-rep i (c)) <Ti } 
else 
W; := irii U {m} 
if primaryii) ^ « then 
owfi := outi U {m} 

RECEIVE((PRE-PREPARE, v, n,m) aj )i (j ^ i) 
Eff: if j = primaryii) A in-v(v, i) A /ia5-neH'-vfew(u, «)A 

53rf : (d ^ D(m) A (PREPARE, u, n, d, i),^ 6 f'n;) then 
letp = (PREPARE, v, n, D(m),i) (7i 
irii := irii U { (PRE-PREPARE, v, n, m)^ ,p} 
outi := outi U {p} 
else if 3o, t,c : (m = (REQUEST, o, t, c) CT< .) then 
irii '■= irii U {m} 

RECEIVE( (PREPARE, V, n, d, j) aj ), ( j ^ i) 
Eff: if j ^ primaryii) A f'n-v(u, «) then 

mi := in; U {(PREPARE, v, n, d, j) a . } 

RECEIVE( (COMMIT, v, n, d, j) crj ), (j ^ «) 
Eff: if viewi > u then 

in; := j«i U {(COMMIT, w, n, d, j)^ . } 

RECEIVE( (VIEW-CHANGE, V,V, j) aj )i U ^ i) 
Eff: let m = ( VIEW-CHANGE, v,V,j) l r i 

if v > viewi A correct-view-changeim,v, j) then 
(«; := irii U {m} 

receive( (new- VIEW, V, X, O, N) <Tj )i (j ^ «) 
Eff: let m = (new- view, v,X,0,N) aj , 

P = {(prepare, v, n', Dim'), i) tTi | (PRE-PREPARE, v, n',m') <Tj 6 (O U N)} 
if v > A v > viewi A correct-new-viewim, v) A -ihas-new-viewiv, i) then 
viewi := v 

irii := iriiUOUNU {m} U P 
outi := -P 

REPLICA-FAILURE; 
Eff: faulty i := true 



Output Transitions: 

SEND(m, TZ-^{i})i 
Pre: m € outi A ^tag(m, REQUEST) A ^tag(m, REPLY) 
Eff: outi := outi -S>{m} 

SEND(m, {primary(i)})i 
Pre: m 6 outi A tag(m, REQUEST) 
Eff: outi := outi •^■{m} 

SEND((REPLY, v, t, c, i, r) CT; , {c})i 
Pre: (REPLY, v,t,c, i,r) ai € outi 
Eff: outi := ouh <»{ (REPLY, v, t, c, i, r) ai } 

Internal Transitions: 

SEND-PRE-PREPARE(m, V, n), 
Pre: primary(i) = i A seqno i =n#lA in-v(v, «) A /los-new-view^, «)A 

3o, t,c : (m= (REQUEST, o, t, c) ac Am 6 W;)A j2(PRE-PREPARE, v, n',m) l7i 6 in; 
Eff: seqnOi := seqno i + 1 

let p = (PRE-PREPARE, v, n, m),^ 
owfi := ot(fi U {p} 
im := hi U {p} 

SEND-COMMIT(m, V, n), 
Pre: prepared(m, v, n, i) A (COMMIT, v, n, D(m),i) ai £ irii 
Eff: let c = (COMMIT, v, n, D(m),i) <Ti 
outi := outi U {c} 
irii := f«; U {c} 

EXECUTE(m, v,n)i 
Pre: n = last-exea + 1 A committed(m, v, n, i) 
Eff: last-exec i := n 
if (m ^ raw//) then 
let (REQUEST, o, i, c) (Tc = m 
if t > last-rep-ti(c) then 
if t > last-rep-t^c) then 
last-rep-t^c) := t 
(last-rep ^c), vak) := g(c, o, va/i) 
oafi := OKti U {(REPLY, view;, t, c, i, last-rep i (c)) ai } 
j'Mi := m; <»{m} 

SEND-VIEW-CHANGE(u) i 
Pre: v = viewi + 1 
Eff: viewi := i> 

let P' = {(m,v,n)\last-prepared(m,v,n,i)}, 
P = U< m ,„,„>gP' ({P = (PREPARE, v, n, D(m), h) <Tk \p E im} U { (PRE-PREPARE, v, n, m) aprimaryM }), 
m = (VIEW-CHANGE, V, P, i) cri 
outi '■= outi U {m} 
irii '■= irii U {m} 
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SEND-NEW- VIEW(v, V)i 
Pre: primary(v) = i A v > viewi A v > Q AV ^ irii A \V\ = 2/ + 1 A -ihas-new-view(v, i)A 

3R : (\R\ = 2/ + 1 A Vfc G R : (3P : (( VIEW-CHANGE, v, P, k) ak 6 V))) 
Eff: viewi := v 

let O = { (PRE-PREPARE, v, n, m) <Ti \3v' : last-prepared(m, v' , n, merge-P(V))} , 
N = { (PRE-PREPARE, v, n, null, k) ai \n < max-n(0)A 3v' , m, n : last-prepared(m, v' , n, merge-P(V))} , 
m = (new- view, v, V, O, N) (Ti 
seqno i := max-n{0) 
hi := trull Oil Nil {m} 
outi := {m} 



4.4 Safety Proof 

This section proves the safety of our algorithm, i.e., it proves that A implements S. We start by 
proving some invariants. 

Invariant 4.1 The following is true of any reachable state in an execution of A, 

V i,j e 11, m e M : ({-^faulty { A -ifaulty J A -*tag(m, REPLY)) => 

{{{m) <Ti 6 irij V 3 m' = (VIEW-CHANGE, v, P, k) ak : (m' 6 irij A {m) <Ti G P) V 
3 m' = (NEW-VIEW, v, V, O, N)v k : (m' 6 irij A {{m) ai 6 V V {m) ai 6 merge-P(V)))) 

=>• {m) <Ti e irii)) 

The same is also true if one replaces irij by {m \ 3X : (m, X) G wire} or by outj 

Proof: For any reachable state x of A and message value m that is not a reply message, if replica i 
is not faulty in state x, (m) (Ti 6 outi =>• ( rn )<r i G ini. Additionally, if (m) <Ji G in* is true for some 
state in an execution, it remains true in all subsequent states in that execution or until i becomes 
faulty. By inspection of the code for automaton Ri , these two conditions are true because every 
action of Ri that inserts a message (m) (Ti in outi also inserts it in ini and no action ever removes a 
message signed by i from im. 

Our assumption on the strength of authentication guarantees that no automaton can imper- 
sonate a non-faulty replica Ri by sending (m) (Ti (for all values of m) on the multicast channel. 
Therefore, for a signed message (m) (Ti to be in some state component of a non-faulty automaton 
other than Ri, it is necessary for SEND({m) (Ti , X)i to have executed for some value of X at some 
earlier point in that execution. The precondition for the execution of such a send action requires 
(m) (Ti G outi. The latter and the two former conditions prove the invariant. Q 



Invariant 4.2 The following is true of any reachable state in an execution of A, for any replica i 
such that faulty i is false: 

1. V(PREPARE, v, n, d, i) tTi G m : (3d' ^ d : ((PREPARE, v,n,d', i) <Ti E in,)) 

2. Vv,n,m : ((i = primary(v) A (PRE-PREPARE, v,n, m) l7i 6 im) => 
3m' : {ml ^ m A (PRE-PREPARE, v,n, m!) cri 6 im)) 

3. V(PRE-PREPARE, v,n, m) ai € irii : (i = primaryiv) => n < seqriOj) 

4. V{PRE-PREPARE,v,n,m) aprimary(v) 6 im : 

(v > => 3m' = (new-view, v, X, O, N) <Tpr i mary : (m' 6 m A correct-new-view {m! , v))) 
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5. Vm' = (NEW- VIEW, v, X, O, N)v vr i mary , € iiti : correct-new-view(m' ,v) 

6. Vm' = (VIEW-CHANGE, v, V, j)& ■ € fn; : correct-view-change(m' ,v,j) 

7. V(prepare, u,n, D(m),i) ai 6 in, : ((pre-prepare, u,n, m) (T p rimar y {v) E im) 

8. V(PRE-PREPARE, u,n, m) <Tpn - mar y (j)) 6 im : (i j= primary(v) =>• (PREPARE, v,n, D(m),i) <Ti G im) 

9. V(PRE-PREPARE, i),n, m)o-„ n - mflry , . 6 rn; : v < viewt 

Proof: The proof is by induction on the length of the execution. The initializations ensure that 
itii = {} and, therefore, all conditions are vacuously true in the base case. For the inductive step, 
assume that the invariant holds for every state of any execution a of length at most /. We will 
show that the invariant also holds for any one step extension a^ of a. 

Condition (1) can be violated in a.\ only if an action that may insert a prepare message signed 
by i in im executes. These are actions of the form: 

1. receive( (pre-prepare, V, n, m') CTj )i 

2. receive( (prepare, V, n, d, j) a - )i 

3. receive( (new- VIEW, V, V, O, N) aj ); 

The first type of action cannot violate condition (1) because the condition in the if 
statement ensures that (prepare, v,n, D{m'),i) ai is not inserted in im when there exists a 
(prepare, v,n, d,i) (Ti 6 im such that D(m') / d. Similarly, the second type of action can- 
not violate condition (1) because it only inserts the argument prepare message in im if it is signed 
by a replica other than Ri . 

For the case v = 0, actions of type 3 never have effects on the state of Ri . For the case v > 0, 
we can apply the inductive hypothesis of conditions (7) and (4) to conclude that if there existed a 
(prepare, v, n, D(m),i) (Ti 6 im in the last state in a, there would also exist a new-view message 
for view v in im in that state. Therefore, the precondition of actions of type 3 would prevent 
them from executing in such a state. Since actions of type 3 may insert multiple prepare messages 
signed by Ri into im, there is still a chance they can violate condition (1). However, this cannot 
happen because these actions are enabled only if the argument new-view message is correct and 
the definition of correct-new-view ensures that there is at most one pre-prepare message with a 
given sequence number in O U N. 

Condition (2) can be violated in a^ only by the execution of an action of one of the following 
types: 

1. receive( (pre-prepare, v, n, m') CTj )i, 

2. receive( (new- view, v, V, O, N) aj ) ; , 

3. SEND-PRE-PREPARE(m, V, n)i, or 

4. SEND-NEW-VIEW(u,l/)i 

Actions of the first two types cannot violate condition (2) because they only insert pre-prepare 
messages in im that are not signed by Ri. Actions of the third type cannot violate condition (2) 
because the inductive hypothesis for condition (3) and the precondition for the send-pre -prepare 
action ensure that the pre-prepare message inserted in im has a sequence number that is one higher 
than the sequence number of any pre-prepare message for the same view signed by Ri in im. 

12 



Finally, actions of the fourth type cannot violate condition (2). For v = 0, they are not enabled. 
For v > 0, the inductive hypothesis of condition (4) and the precondition for the send-new- view 
action ensure that no pre -prepare for view v can be in im when the action executes, and the 
definition of O and N ensures that there is at most one pre-prepare message with a given sequence 
number in O U N. 

Condition (3) can potentially be violated by actions that insert pre-prepares in im or modify 
seqnoi- These are exactly the actions of the types listed for condition (2). As before, actions of the 
first two types cannot violate condition (3) because they only insert pre-prepare messages in im 
that are not signed by Ri and they do not modify seqno^. The send-pre -prepare action preserves 
condition (3) because it increments seqno i such that it becomes equal to the sequence number of 
the pre-prepare message it inserts in im. The send-new-view actions also preserve condition (3): 
(as shown before) actions of this type only execute if there is no pre-prepare for view v in im and, 
when they execute, they set seqno i := max-n(0), which is equal to the sequence number of the 
pre-prepare for view v with the highest sequence number in im- 

To violate condition (4), an action must either insert a pre-prepare message in im or remove a 
new-view message from im ■ No action ever removes new-view messages from im . The actions that 
may insert pre-prepare messages in im are exactly the actions of the types listed for condition (2). 
The first type of action in this list cannot violate condition (4) because the if statement in its body 
ensures that the argument pre-prepare message is inserted in im only when has-new-view(v, i) is 
true. The second type of action only inserts pre-prepare messages for view v in im if the argument 
new-view message is correct and in this case it also inserts the argument new-view message 
in im. Therefore, the second type of action also preserves condition (4). The precondition of 
send-pre-prepare actions ensures that send-pre -prepare actions preserve condition (4). Finally, 
the send-new-view actions also preserve condition (4) because their effects and the inductive 
hypothesis for condition (6) ensure that a correct new- view message for view v is inserted in im 
whenever a pre-prepare for view v is inserted in im. 

Conditions (5) and (6) are never violated. First, received new-view and view-change messages 
are always checked for correctness before being inserted in im. Second, the effects of send-view- 
change actions together with the inductive hypothesis of condition (9) and the precondition of 
send-view-change actions ensure that only correct view-change messages are inserted in im. 
Third, the inductive hypothesis of condition (6) and the effects of send-new-view actions ensure 
that only correct new-view messages are inserted in im. 

Condition (7) is never violated because no action ever removes a pre-prepare from im 
and the actions that insert a {prepare, v, n, D(m),i) ai in im (namely actions of the form 
RECEIVE( (PRE-PREPARE, t>, n,m') a:j )i and RECEiVE( (NEW- VIEW, v, V, 0,N) a:j )i) also insert a 
(pre-prepare, v, n, m) <7primary(v) in im. 

Condition (8) can only be violated by actions that insert pre-prepare messages in im because 
prepare messages are never removed from im ■ These are exactly the actions listed for condition (2). 
The first two types of actions preserve condition (8) because whenever they insert a pre-prepare 
message in im they always insert a matching prepare message. The last two types of actions can 
not violate condition (8) because they never insert pre-prepare messages for views v such that 
primary(v) ^ i in im. 
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The only actions that can violate condition (9) are actions that insert pre-prepare messages in 
itii or make viewi smaller. Since no actions ever make viewi smaller, the actions that may violate 
condition (9) are exactly those listed for condition (2). The j/ statement in the first type of action 
ensures that it only inserts pre-prepare messages in im when their view number is equal to viewi. 
The if statement in the second type of action ensures thatv it only inserts pre-prepare messages 
in im when their view number is greater than or equal to viewi. Therefore, both types of actions 
preserve the invariant. The precondition for the third type of action and the effects of the fourth 
type of action ensure that only pre-prepare messages with view number equal to viewi are inserted 
in im. Thus, these two types of actions also preserve the invariant. □ 

Definition 4.3 n-faulty = \{i € 7^ | faulty ^ = true}| 

Invariant 4.4 The following is true of any reachable state in an execution of A, 

Vi, j e 11, n, v e N, m,m' e M : ((-faulty \ A -faulty ■• A n-faulty < /) =>- 
(prepared(m,v,n,i) A prepared(m! ,v,n,j) =>• D(m) = D(m'))) 



Proof: By contradiction, assume the invariant does not hold. Then prepared(m, v, n, i) = true 
and prepared(m' , v, n, j) = true for some values of m, m', v, n, i, j such that D(m') ^ D(m). 
Since there are 3/ + 1 replicas, this condition and the definition of the prepared predicate imply: 

(a)3R: (\R\ > f A VJfc £ R : 

((((PRE-PREPARE, v,n,m) (Th 6 im A k = primary(v)) V (PREPARE, v,n, D(m),h) (Th 6 im) A 
(((PRE-PREPARE, u,n, m%,. £ irij A k = primary (v)) V (PREPARE, v, n, D(m'), A%, € inj))) 

Since there are at most / faulty replicas and R has size at least / + 1, condition (a) implies: 

(b) 3k g R : (faulty k = false A 

((((PRE-PREPARE,7J, n, 771)0-, e im A k = primary (v)) V (PREPARE, v, n, D(m), A%, € im) A 
(((PRE-PREPARE, ?j,7i,m%, e itij A k = primary(v)) V (PREPARE, v, n, D(m'), A%, 6 inj))) 

Invariant 4.1 and (b) imply: 

(c) 3k G R : (faulty k = false A 

((((PRE-PREPARE,7J, 7i, 771)0-, e in k A k = primary(v)) V (PREPARE, v, n, D(m), A%, e ink) A 
(((PRE-PREPARE, 7j,77,m')o, G wijt A k = primary (v)) V (PREPARE, v, n, D(m'), k) ah G im))) 

Condition (c) contradicts Invariant 4.2 (conditions 1, 7 and 2.) □ 

Invariant 4.5 The following is true of any reachable state in an execution of A, 

Vi 6 R : ((-^faulty i A n-faulty < f) =>• 
(V (NEW- VIEW, u, 1/, O, A%, G m, n,v' G N : 
(prepared(m, v' ,n,merge-P(V)) A prepared(m' ,v' ,n, merge- P(V)) => D(m) = D(m')))) 



Proof: Since Invariant 4.2 (condition 6) ensures any new-view message in im for a non-faulty i sat- 
isfies correct-new-view, the proof for Invariant 4.4 can also be used here with minor modifications. 

D 
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Invariant 4.6 The following is true of any reachable state in an execution of A, 

V« € 1Z : (-faulty i =>• V(C0MMIT, v, n, d, i) (7i € itii : (3m : (D(m) = d A prepared(m,v,n,i) = true)) 

Proof: The proof is by induction on the length of the execution. The initializations ensure that 
itii = {} and, therefore, the condition is vacuously true in the base case. For the inductive step, 
the only actions that can violate the condition are those that insert commit messages in im, i.e., 
actions of the form RECEIVE( {COMMIT, v, n, d, j) a )i or SEND-COMMlT(m, v, n)j. Actions of the 
first type never violate the lemma because they only insert commit messages signed by replicas 
other than Ri in im. The precondition for send-commit actions ensures that they only insert 
(COMMIT, v, n, D(m),i) (Ti in im if prepared(m, v, n, i) is true. D 

Invariant 4.7 The following is true of any reachable state in an execution of A, 

V« £ 1Z, n,v € N, m € M : ((-<faulty i A committed(m,v,n,i)) => 

(3 R : (\R\ > 2/ •^■n-faulty A Vfc € R : (faulty k = false A prepared(m,v,n,k))))) 



Proof: From the definition of the committed predicate committed(m, v, n, i) = true implies 

(a)3R: (\R\ > 2/ + 1 A VA: e R : ({commit, v,n,D(m),k) (Tk e im)). 

Invariant 4. 1 implies 

(b) 3R : (\R\ > 2/ <$n-faulty A VA: G R : (faulty k = false A (COMMIT, v, n, D(m),k) (7k <G ink)). 

Invariant 4.6 and (b) prove the invariant. D 



Invariant 4.8 The following are true of any reachable state in an execution of A, for any replica 
i such that faulty i is false: 

1. \/m,v,n,P : (( VIEW- CHANGE, v, P, i) ai G im => 

Vv' < v : (last-prepared-b(m,v' ,n,i,v) O last-prepared(m,v' ,n, P))) 

2. Vm = (NEW-YKW,v,V,0,N) aprirmTyM G m : ((O U N) C im) 

Where last-prepared-b is defined as follows: 
last-prepared-b(m, v,n,i,b) = v < b A prepared(m,v,n, /n,)A 

im',v' : ((prepared(m' ,v' ,n, itii) A v < v' < b) V (prepared(m! ,v,n, itii) A m ^ ti')). 

Proof: The proof is by induction on the length of the execution. The initializations ensure that 
m i = {} an d> therefore, the condition is vacuously true in the base case. 

For the inductive step, the only actions that can violate condition (1) are those that insert 
view-change messages in im and those that insert pre-prepare or prepare messages in im (no pre- 
prepare or prepare message is ever removed from im.) These actions have one of the following 
schemas: 

1. receive( (view-change, V, P, j) crj )i 

2. VIEW-CHANGE(u)i 

3. receive((pre-prepare, V, n, m') <Tj )i, 

4. receive( (prepare, V, n, d, j) <Tj )i, 
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5. receive( (new- view, v, V, O, N) aj );, 

6. SEND-PRE-PREPARE(m, V, n)i, or 

7. SEND-NEW-VIEW(v, V), 

Actions of the first type never violate the lemma because they only insert view-change mes- 
sages signed by replicas other than Ri in im. The effects of actions of the second type ensure that 
when a view-change message (VIEW-CHANGE, v, P, i) <Ji is inserted in im the following condition 
is true: 

(a) W < v : (last-prepared(m, v', n, i) «=> last-prepared(m, v', n, V)). Condition (a) and Invari- 
ant 4.2 (condition 9) imply condition 1 of the invariant. 

For the other types of actions, assume there exists at least a view change message for v signed 
by Ri in in% before one of the other types of actions executes (otherwise the lemma would be 
vacuously true) and pick any m' = (VIEW-CHANGE, v, P,i) ai £ i n i- The inductive hypothesis 
ensures that the following condition holds before the actions execute: 

Vm, n, v 1 < v : (last-prepared-b(m, v' , n, i, v) 4^ last-prepared(m, v' , n, V)) 

Therefore, it is sufficient to prove that the actions preserve this condition. The logical value 
of last-prepared(m,v' ,n,V)) does not change (for all m',m,n,v') because the view-change 
messages in im are immutable. 

To prove that the value of last-prepared-b(m, v', n, i, v) is also preserved (for all m', m, n, v'\ 
we will first prove the following invariant (b): For any reachable state in an execution of A, any 
non-faulty replica Ri, and any view-change message m' = (VIEW-CHANGE, v, P, i) ai , m' 6 im =$■ 
viewi > v. 

The proof for (b) is by induction on the length of the execution. It is vacuously true in the base 
case. For the inductive step, the only actions that can violate (b) are actions that insert view-change 
messages signed by Ri in im or actions that make viewi smaller. Since there are no actions that 
make viewi smaller, these actions have the form VIEW-CHANGE (u) $. The effects of actions of this 
form ensure the invariant is preserved by setting viewi to the view number in the view-change 
message. 

Given (b) it is easy to see that the other types of actions do not violate condition 1 of the 
lemma. They only insert pre -prepare or prepare messages in im whose view number is equal to 
viewi after the action executes. Invariant (b) guarantees that viewi is greater than or equal to the 
view number v of any view-change message in im ■ Therefore, these actions cannot change the 
value of last-prepared-b(m, v' , n, i, v) for any m', m, n, v'. 

Condition (2) of the lemma can only be violated by actions that insert new-view messages in 
im or remove pre-prepare messages from im. Since no action ever removes pre-prepare messages 
from im, the only actions that can violate condition (2) are: RECEIVE( (new- view, v, V, O, N) a )i 
and SEND-NEW- view(u, V)i. The first type of action preserves condition (2) because it inserts all 
the pre -prepares in O U N in im whenever it inserts the argument new-view message in im. The 
second type of action preserves condition (2) in a similar way. □ 



Invariant 4.9 The following is true of any reachable state in an execution of A, 
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V« € 1Z, m G M., v,n G N : ((-faulty i A n-faulty < /A 
3 7? : (|72| > / A V fc G -R : (-^faulty k A prepared(m,v,n,k))) =>• 

Vt/ > u G N, m' 6 X : ((PRE-PREPARE, i/,ra, m')^ nnM , } G in< =>• m' = m)) 

Proof: Rather than proving the invariant directly, we will prove the following condition is true: 

V« G 7Z, m G jM, u,n G N : ((-^faulty { A n-faulty < / A 
3fl : (|i?| > / A V fc G -R : (-^faulty k A prepared(m,v,n,k))) => 
Vt/ > « e N, (NEW-VIEW, u',1/,0, TV) a primamvl) G Wi : 
((PRE-PREPARE,t;',n,m) CTpnmao;(t) , ) G O)) 

Condition (a) implies the invariant. Invariant 4.2 (condition 4) states that there is never a 
pre-prepare message in im for a view v' > without a correct new-view message in in* for the 
same view. But if there is a correct new-view message (new- VIEW, v', V, O, ^}(T Pr i mar y (vl) £ Wi 
then Invariant 4.8 (condition 2) implies that (O U N) C inj. This and condition (a) imply that there 
is a (pre-PREPARE, ?/,n, m}^ mfl , E im and Invariant 4.2 (conditions 1,2 and 8) implies 
that no different pre-prepare message for sequence number n and view v' is ever in im. 

The proof is by induction on the number of views between v and v'. For the base case, 
v = v', condition (a) is vacuously true. For the inductive step, assume condition (a) holds for v" 
such that v < v" < v'. We will show that it also holds for v'. Assume there exists a new-view 
message mi = (new- VIEW, v', V\,0\, N\) a V rimarv( v '\ m JW * (otherwise (a) is vacuously true.) 
From Invariant 4.2 (condition 5), this message must verify correct-new -view{m\, v'). This implies 
that it must contain 2/ + 1 correct view-change messages for view v' from replicas in some set 
Rl 

Assume that the following condition is true (b) 3R : (\R\ > f A Vfc e R : (faulty k = false A 
prepared(m,v,n, k) = true)) (otherwise (a) is vacuously true.) Since there are only 3/ + 1 replicas, 
R and R\ intersect in at least one replica and this replica is not faulty; call this replica k. Let fc's 
view-change message in m\ be mi = (VIEW-CHANGE, v', Pi, k) ak . 

Since k is non-faulty and prepared(m, v, n, k) = true, Invariant 4.4 implies that 
last-prepared-b(m, v, n, k, v + 1) is true. Therefore, one of the following conditions is true: 

1. last-prepared-b(m,v,n, k, v') 

2. 3v",m' : (v < v" < v' A last-prepared-b(m! ,v" ,n,k,v')) 

Since condition (a) implies the invariant, the inductive hypothesis implies that m = m' in the 
second case. Therefore, Invariants 4. 1 and 4.8 imply that (c) 3v 2 > v : last-prepared(m, v 2 ,n, P 2 ) 

Condition (c), Invariant 4.5, and the fact that correct-new -view[m\, v') is true imply that one 
of the following conditions is true: 

1. last-prepared(m, v 2 ,n, merge-P{V\)) 

2. 3v",m' : (v 2 < v" < v' A last-prepared(m' ,v" ,n,merge-P(Vi))) 

In case (1), (a) is obviously true. If case (2) holds, Invariant 4.1 and Invari- 
ant 4.2 (condition 7) imply that there exists at least one non-faulty replica j such that 
(pre-PREPARE, v",n,m') (T i ma „ E inj. Since condition (a) implies the invariant, the in- 
ductive hypothesis implies that m = m' in the second case. □ 
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Invariant 4.10 The following is true of any reachable state in an execution of A, 

V n, v, v' G N, m,m' G M. : (n-faulty < f =>- 

(3 R C TZ : (\R\ > f A V k G R : (->faulty k A prepared(m,v,n,k))) A 

3E' C TZ : (\R'\ > f A V k G R' : (-^faulty k A prepared{m' ,v' ,n,k)))) => D(m) = D{m!)) 



Proof: Assume without loss of generality that v < v' . For the case v = v' , the negation 
of this invariant implies that there exist two requests m and m' (D(m') / D{m)), a se- 
quence number n, and two non-faulty replicas Ri,Rj, such that prepared(m,v,n,i) = true 
and prepared(m' , v, n, j) = true; this contradicts Invariant 4.4. 

For v > v', assume this invariant is false. The negation of the invariant and the definition of 
the prepared predicate imply: 

3 n, v, v' G N, m, m! G M. : (v > v' A n-faulty < / A 

(3 R C TZ : (\R\ > f A V k G R : (->faulty k A prepared(m,v,n,k))) A 

3i £ K : (-^faulty { A (pre-prepare,^,^™'),^-^^^ G mi) A .D(m) ^ D(m')) 

But this contradicts Invariant 4.9 as long as the probability that m ^ vn! while D(m) = D(m') 
is negligible. Q 



Invariant 4.11 The following is true of any reachable state in an execution of A, 

V«,j G TZ, n,v,v' G N, m, m! G M : {{-<faulty i A -faulty A n-faulty < /) =>- 
(committed(m,v,n,i) A committed(m' ,v' ,n,j) =$> D(m) = D(m'))) 



Invariant 4.12 The following is true of any reachable state in an execution of A, 

V« G TZ, n,v,v' G N, m,m' G .M : ((-faulty i A n-faulty < /) =>• (committed(m,v,n,i) A 

3fl' CK : (|,R' | > / A Vfc G E' : (-^faulty k A prepared(m' ,v' ,n,k)))) => D(m) = D(m')) 



Proof: Both Invariant 4. 1 1 and 4. 12 are implied by Invariants 4. 10 and 4.7. □ 

Rather than proving that ^4 implements 5 directly, we will prove that A implements S", which 
implements S and is better suited for the proof. We start by defining a set of auxiliary functions 
that will be useful for the proof. 

Definition 4.13 We define the following functions inductively: 

val : (N x O 1 x N x C)* -s- V 

last-rep : (N x O 1 x N x C)* -)■ ( C -)■ 0') 

last-rep-t : (N x 0' x N x C)* -5- ( C -5- N) 

va/(A) = i> 

V c : (last- rep (X)(c) = null-rep) 

Vc : (last-rep-t(\)(c) = 0) 

val(ii.{n,o,t,c)) = s 
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last-rep(p.(n,o,t,c))(c) = r 
last-rep-t(p.(n,o,t,c))(c) = t 

V c' ^ c : (last-rep(p.(n,o,t,c))(c') = last-rep(p)(c')) 

V c 1 ^ c : (last-rep-t(p.(n,o,t,c))(c') = last-rep-t(p)(c')) 
where (r, s) = g(c,o,val(p)) 



Automaton S' has the same signature as S except for the addition of an internal action 
execute-null. It also has the same state components except that the vol component is replaced by 
a sequence of operations: 

hist E (N x O' x N x C)*, initially A; 
and there is a new seqno component: 
seqno E N, initially 0. 

Similarly to S, the transitions for S' are only defined when n-faulty < f. Also, the transitions 
for S" are identical to S"s except for those defined bellow. 



execute(o, t,c) 
Pre: (o, t,c) e in 
Eff: seqno := seqno + 1 
in := in-$${(o, t,c)} 
if t > last-rep-t(hist)(c) then 
fcf := hist. (seqno, o, t, c) 
out := OMf U {(last-rep(c),t, c)} 



EXECUTE-NULL 

Eff: seqno := seqno + 1 



The EXECUTE-NULL actions allow the seqno component to be incremented without removing 
any tuple from in. This is useful to model execution of null requests. 

Theorem 4.14 S' implements S 

Proof: The proof uses a forward simulation [8] T from S' to S. T is defined as follows: 

Definition 4.15 T is a subset of states(S") x states(S'); (x, y) is an element of T (also written as 
y E ^"[a;]j if and only if all the following conditions are satisfied: 

1. All state components with the same name are equal in x and y. 

2. x.val = val(y.hisi) 

3. x.last-rep-t c = last-rep(y .hisi)(c) ,Vc £ C 



To prove that T is in fact a forward simulation from S' to S one most prove that both of the 
following are true [8]. 

1. For all x E start(S'), F[x] f] start(S) / {} 

2. For all (x, n, x') E trans(S'), where x is a reachable state of S' , and for all y E T[x\, where 
y is reachable in S, there exists an execution fragment a of S starting with y and ending 
with some y' E F[x'] such that trace(a) = trace(n). 
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It is clear that T verifies the first condition because all variables with the same name in S and 
S' are initialized to the same values and, since hist is initially equal to A, x.val = v = val(X) and 
x.last-rep-t c = = last-rep(\)(c). 

We use case analysis to show that the second condition holds for each n 6 acts(S'). For 
all actions n except EXECUTE-NULL, let a consist of a single ir step. For ir = EXECUTE-NULL, 
let a be A. It is clear that this satisfies the second condition for all actions but EXECUTE. For 
7r = EXECUTE(o, t, c), definition 4.13 and the inductive hypothesis (i.e., x.val = val(y.hist) and 
x.last-rep-t c = last-rep(y .hist)(c)) ensure that y' 6 JFfic']. □ 



Definition 4.16 We define the function prefix : (NxO'xNxC)* -> (NxO'xNxC)* as follows: 
prefix(/j,, n) is the subsequence obtained from \x by removing all tuples whose first component is 
greater than n. 

Invariant 4.17 The following is true of any reachable state in an execution of S 1 , 

V (n,o,t,c) £ hist : (t > last-rep-t(prefix(hist, n <$=>!)) (c)) 



Proof: The proof is by induction on the length of the execution. The initial states of S' verify 
the condition vacuously because hist is initially A. For the inductive step, the only actions that 
can violate the invariant are those that modify hist, i.e., EXECUTE(o, t, c). But these actions only 
modify hist if t > last-rep-t(hist)(c). □ 



Invariant 4.18 The following are true of any reachable state in an execution of S': 

1. V(n, o, t, c) 6 hist : (-ifaulty c =>• t < last-req c ) 

2. V(o, t, c) £ in : (-ifaulty c => t < last-req c ) 



Proof: The proof is by induction on the length of the execution. The initial states of 5' verify 
the condition vacuously because hist is initially A and in is empty. For the inductive step, since 
no action ever decrements last-req c or changes faulty c from true to false, the only actions that 
can violate the invariant are those that append tuples from a non-faulty client c to hist, i.e., 
EXECUTE(o, t, c) or to in, REQUEST(o, c). The EXECUTE actions only append a tuple (n, o, t, c) to 
hist if (o, t, c) 6 in; therefore, the inductive hypothesis for condition 2 implies that they preserve 
the invariant. The REQUEST actions also preserve the invariant because the tuple (o, t, c) inserted 
in in has t equal to the value of last-req c after the action executes. □ 



Theorem 4.19 A implements S 

Proof: We prove that A implements S', which implies that A implements S (Theorem 4.14.) The 
proof uses a forward simulation Q from A' to S' (A' is equal to A but with all output actions not 
in the external signature of S hidden.) Q is defined as follows. 

20 



Definition 4.20 Q is a subset of states(A') x states(<S"); (x, y) is an element of Q if and only if the 
following are satisfied: 

1. Vi € 1Z : (x.faulty i = y. faulty-replica^ 

2. Vc € C : (x.faulty c = y. faulty- client c ) 

and the following are satisfied when n-faulty < f 

3. Vc G C : (-ix.faulty c => x.last-req c = y.last-req c ) 

4. V« G 1Z : (^x.faulty { =>• x.last-exed < y.seqno) 

5. Vi € 1Z : {->x.faulty i =>• x.vaU = val(prefix(y.hist,x.last-execi))) 

6. \fi € TZ : {->x.faulty i =>• Vc G C : (x. last-rep ^c) = last-rep(prefix(y.hist,x.last-execi))(c))) 

7. V« G 7?. : {-ix.faulty l =>• Vc G C : (x.last-rep-t^c) = last-rep-t(prefix(x.hist,y.last-execi))(c))) 

8. V0 < n < y.seqno : (3(n, o, f , c) G j/.&wf : (37? C 7£, u G N : (|7?| > 2/ ^-y. n-faulty A 

Vfc G 7? : (-^x.faulty k A prepared^ (REQUEST, o, f,c) CTc , u,n, A'. fc)))) 

V 37? C 7£, v, i G N, o G £>, c G C : (|7?| > 2/ -^y. n-faulty At < last-rep-t(prefix(y.hist, n «l))(c)) A 

Vfc G 7? : (-^xfaulty k Aprepared((REQUEST,o,t,c) crr ,v,n,A'.k)))) 

V 37? C 7£, v G N : (|7?| > 2/ <&y .n-faulty A Vfc G 7? : (->x.faulty k A prepared({null, v, n, A'.k)))) 

9. V (REPLY, u,£,c, i, r) CTi G (x.outi U {m|3X : (m, X) G x.wire} U x.in c ) : 

(~ix.faulty i =>• 3(n, o, t,c) G y.hist : (r = last-rep(prefix(y.hist,n))(c))) 

10. \/(n,o,y.last-req c ,c) € y.hist: 

((-ix.faulty c Ax.out c ^ {}) => 3(last-rep(prefix(y.hist,n))(c),y.last-req c ,c) G y.owf) 

11. Let M c = x.outc U {m|3« G 7?. : (->x.faulty i Am € x.ini U x.ouh} U {m|3X : (m,X) G x.wire}, 
and M c ' = merge-P({m = (VIEW-CHANGE, u, P, j) aj \m G M c V 

3{NEW-vmw,v,V,0,N) aj G M c : (m G V)}), 
Vc G C : (->x.faulty c =>VoGO,(eN: ((m = (REQUEST, o, f , c) CTc G M C V 

3(PRE-PREPARE, «), n,m) <Tj G M c U M c ') =>■ ({o,t,c) G t/.i«V 3n : ((n,o,t,c) G y.hist)))) 



Note that most of the conditions in the definition of Q only need to hold when n-faulty < f, 
for n-faulty > f any relation will do because the behavior of S' is unspecified. To prove that Q is 
in fact a forward simulation from A! to S' one most prove that both of the following are true. 

1. For all x G start(A'), Q[x\ n start(S') / {} 

2. For all (x, n, x') 6 trans(A'), where a; is a reachable state of A', and for ally 6 Q[x], where 
y is reachable in S', there exists an execution fragment a of 5' starting with y and ending 
with some y' € £7 [a;'] such that frace(a) = trace(n). 

It is easy to see that the first condition holds. We use case analysis to show that the second 
condition 2 holds for each n 6 acts(A') 

Non-faulty proxy actions. If n = REQUEST(o) c , it = CLIENT-FAILURE,,, or ir = REPLY(r) c , 
let a consist of a single n step. Q is preserved in a trivial way if n is a CLIENT-FAILURE action. 
If 7r is a REQUEST action, neither n nor a modify the variables involved in all conditions in the 
definition of Q except 3, and 10 and 11. Condition 3 is preserved because both n and a increment 
y.last-req c . Condition 10 is also preserved because Invariant 4.18 implies that there are no tuples 
in y.hist with timestamp y'.last-req c and a does not add any tuple to y.hist. Even though n inserts 
a new request in x.out c , condition 1 1 is preserved because a inserts (o, t, c) in y.in. 
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If 7r is a REPLY(r) c action that is enabled in x, the REPLY(r) c action in a is also enabled. Since 
there are less than / faulty replicas, the precondition of n ensures that there is at least one non- 
faulty replica i and a view v such that (reply, v, x.last-req c , c, i, r) (Ti 6 x.in c and that x.out c 7^ {}. 
Therefore, the inductive hypothesis (conditions 9 and 10) implies that (r, t, c) 6 y.out and thus 
REPLY(r) c is enabled. Q is preserved because n ensures that x' .out c = {}. 

If 7r = RECElVE(m) c , or 7r = SEND(m, X) c , let a be A. This preserves Q because y G Q[x] 
and the preconditions require that the reply message being received is in some tuple in x.wire and 
the request message being sent is in x.out c . 

Internal channel actions. If tt is a MiSBEHAVE(m, X, X') action, let a be A. Q is preserved 
because n does not add new messages to x.wire and retains a tuple with m on x 1 .wire. 

Non-faulty replica actions. For all actions ir except ir = replica-failurEj and ir = 
EXECUTE(m,'U,n)j, let a be A. It is clear that this could only violate conditions 8, 9 and 11 
because these actions do not modify the state components involved in the other conditions. They 
can not violate condition 8 ; since no messages are ever removed from ink (where k is any non-faulty 
replica), if prepared(m, v, n, k) = true, it remains true for the entire execution or until replica k 
becomes faulty. And these actions do not violate conditions 9 and 1 1 because any request or reply 
messages they add to x.im, x.outi, or x.wire (either directly or as part of other messages) was 
already in x.wire, x.ini, or x.outi. 

For 7r = REPLICA-FAILURE;, let a consist of a single tt step. This does not violate the 
conditions in the definition of Q. For conditions other than 1 and 8, it either does not change 
variables involved in these conditions (2 and 3), or makes them vacuously true. Condition 1 is 
satisfied in a trivial way because a also sets y.faulty-replic^ to true. And condition 8 is not 
violated because the size of the sets R in the condition is allowed to decrease when additional 
replicas become faulty. 

Non-faulty replica execute (non-null request.) 

For 7r = EXECUTE( (request, o, t, c) ac , v, n)i, there are two cases: if x.last-execi < y.seqno, 
let a be A; otherwise, let a consist of the execution of a single EXECUTE(o, t, c) action preceeded 
by FAULTY-REQUEST(o, t, c) in the case where x.faulty c = true. In any of these cases, it is clear 
that only conditions 4 to 1 1 can be violated. 

For the case where a = A, conditions 4, 8, 10 and 11 are also preserved in a trivial way. 
For the other conditions we consider two cases (a) t > last-rep-t^c) and (b) otherwise. The 
precondition of n ensures that x. committed^ (REQUEST, o, t, c) ac , v, n, i) is true. In case (a), this 
precondition, Invariant 4.12, and the definition of Q (condition 8) imply that there is a tuple in 
y.hist with sequence number n and that it is equal to (n, o, t, c). Therefore, conditions 5 to 7 and 
9 are preserved. In case (b), the precondition of n, Invariant 4.12, the definition of Q (condition 
8), and Invariant 4.17 imply that there is no tuple with sequence number n in y.hist. Therefore, 
conditions 5 to 9 are preserved in this case. 

For the case where a / A, when n is enabled in x the actions in a are also enabled in 
y. In the case where c is faulty, FAULTY-REQUEST(o, t, c) is enabled and its execution enables 
EXECUTE(o, t, c). Otherwise, since y 6 Q[x\, condition 1 1 in Definition 4.20 and the precondition 
of ir imply that EXECUTE(o, t, c) is enabled in y. 
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It is easy to see that conditions 4 to 7 and 9 to 1 1 are preserved. For condition 8, we consider 
two cases (a) t > last-rep-t^c) and (b) otherwise. In both cases, the precondition of n ensures 
that x.committed((REQUEST, o, t, c) ac , v, n, i) is true. This precondition, Invariant 4.7 and the fact 
that a appends a tuple (y'.seqno, o,t,c) to y.hist, ensure that condition 8 is preserved in this case. 
In case (b), the precondition Invariant 4.7 and the assumption that t < last-rep-t^c), ensure that 
condition 8 is preserved also in this case. 

Non-faulty replica execute (null request.) 

For 7r = EXECUTE(nu/Z, v, n)i, if x.last-execi < y.seqno, let a be A; otherwise, let a consist 
of the execution of a single EXECUTE-NULL action. Execution of a null request only increments 
x.last-execi and a can at most increment y.seqno. Therefore, only conditions 4 to 8 can be violated. 
Condition 4 is not violated because a increments y.seqno in the case where x.last-execi = y.seqno. 

For the case where, a = X, conditions 5 to 7 are also not violated because a does not append 
any new tuple to y.hist and all tuples in y.hist have sequence number less than y' .seqno; therefore, 
prefix(y.hist, x.last-execi) = prefix(y' .hist, x' .last-execi). Since the precondition of n implies 
that x.committed(null, v, n, i) is true, Invariant 4.7 ensures condition 8 is also preserved in this 
case. 

For the case where a consists of a EXECUTE-NULL step, x.committed(null,v,n,i), 
n-faulty < f, Invariant 4.12, and the definition of Q (condition 8) imply that there is no 
tuple in y' .hist with sequence number x' .last-execf, therefore, prefix(y.hist,x.last-execi) = 
prefix(y' .hist, x' .last -exec i) . 

Faulty replica actions. If n is an action of a faulty replica i (i.e., x.faulty i = true), let a be 
A. Since n can not modify faulty i and a faulty replica cannot forge the signature of a non-faulty 
automaton this preserves Q in a trivial way. 

Faulty proxy actions. If n is an action of a faulty proxy c (i.e., x. faulty c = true), let a consist 
of a single n step for REQUEST, REPLY and CLIENT-FAILURE actions and A for the other actions. 
Since n can not modify faulty c and faulty clients cannot forge signatures of non-faulty automata 
this preserves Q in a trivial way. Additionally, if n is a REPLY action enabled in x, n is also enabled 
iny. D 



5 Garbage Collection 

This section describes a modified version of our algorithm that garbage collects messages from 
replica's logs. It also proves that the modified algorithm A gc is safe, i.e. , it proves that it implements 

S. 



5.1 The Modified Algorithm 

The client proxy and multicast channel automata are identical in A gc and A. The replica automaton 
Ri is modified as follows. The signature remains the same except for the actions listed below. 
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Input: receive( (checkpoint, v, n, d, j) aj )i 

receive( (view-change, v, n, s, C, P, j)<j. ) 
Internal: collect-garbage; 

Here, v, n G N, i, j G 11, s G V, C, P C M, d G V 

where V'=Vx(C->C)x(C->N) and X>' = {d | 3s G V : (d = D(m))} 

The state components also remain the same except for the addition of a new variable chkpts i 
and a new initial value for im : 

itii C .M, initially {(CHECKPOINT, 0, D((v , null-rep, 0)),k) ak | Vfc G 7£} 
chkpts { C N x V', initially {(0, (vo, null-rep, 0))} 
stable-ni = mm({n (n, ^) € chkpts { }) 
stable-chkpt i = v \ (stable-Hi , v) 6 chkpts i 

The auxiliary functions used in the description of a replica's automaton also remain the same 
except for those that are defined below: 

in-w(n, i) = < n •^■stable-ni < max-out, where max-out € N 
in-wv(v, n, i) = in-w(n, i) A in-v(v, i) 

correct-view-change(m,v,j) = 3n,s,C,P: (m = (VIEW-CHANGE, v,n, s, C, P, j) aj A 
3R : (\R\ > / A VA: G R : (3 v" < v : ((CHECKPOINT, v",n, D(s), k)„ k 6 C)) A 
V (PRE-PREPARE, v' ,n' ,m')< Tprimary{v , ) e P : 

(last-prepared(m! , v' , n' , P) A v' < v A < n' •^■n < max-out) 
merge-P(V) = { m \ 3 (VIEW-CHANGE, v, n, s, C, P, k) ak 6 V : (m 6 P) } 

max-n(M) = max({ n \ (PRE-PREPARE, v, n, m) c , i E M V {VIEW-CHANGE, v,n,s,C,P,i) c , i 6 M}) 
correct-new-view(m, v) = 
3V,0,N,R: (m = (mv/-vmw,v,V,0,N)„ primafyM A \V\ = \R\ = 2/ + 1 A 
V k € R : (3 m! € V : (correct-view-change(m' ,v,k))) A 

O = { (PRE-PREPARE, v,n, m') CT „ nmflrv | n > max-n(V) A 3 t/ : last-prepared(m' ,v' ,n,merge-P(V))} A 
JV = { (PRE-PREPARE, u, n, null} ar>r j marv , ) I max-n(V) < n < max-n(0) A 

2v',m',n : last-prepared(m' ,v' ,n, merge-P(V))) 
take-chkpt(n) = (n mod chkpt-int) = 0, where chkpt-int € N A chkpt-int < max-out 
update-state-nv(i, v,V,m) = 
if max-n(V) > stable-rii then 
in; := («i U (pickC : 3 (view-change, v,ma*-w(V), s, C, P, fc),^ G V) 
if (CHECKPOINT, v,max-n(V),D(s),i) (7i # im then 
im = im U {(checkpoint, v,max-n(V),D(s),i) <Ti } 
outi = outi U {(CHECKPOINT, v,max-n(V),D(s),i) ai } 
chkpts i := chkpts i ■& {p = {n',s') \p 6 chkpts i A n' < max-n(V)} 
if max-n(V) > last-execi then 
chkptSi := chkptSi U {{max-n(V), s) \ 3 (VIEW-CHANGE, v,max-w(V), s, C, P, fc) CTil , G V} 
(vah,last-rep i ,last-rep-t i ) := stable-chkpt i 
last-execi := max-n(V) 



Many of the actions for automaton Ri are modified to use the new functions but otherwise 
remain identical. The exceptions are listed below: 
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Input Transitions 

RECEIVE(PRE-PREPARE, U, n,m) <Tj )i (j =£ i) 
Eff: if j = primaryii) A in-wv(v, n, i) A has-new-view(v, i)/\ 
3d : (d j^ D{m) A (PREPARE, v, n, d, i) ai 6 im) then 
letp = (PREPARE, v, n, D(m),i) <Ti 
im := im U { (PRE-PREPARE, v, n, m) rTj ,p} 
ouU := ouu U {p} 
else if 3o, t,c : (m = (REQUEST, o, t, c}^) then 
im := im U {m} 

receive( (prepare, V, n, d, j) crj ), ( j ^ j) 
Eff: if j ^ primaryii) A in-wv(v, n, i) then 
mi := (ft; U {(PREPARE, v, n, d, j) a . } 

RECEIVE ((COMMIT, V, n, d, j) crj ), (j ^ i) 
Eff: if View; >cA in-w(n, i) then 

mi := im U {(COMMIT, w, n, d, j) a . } 

RECEIVE ((CHECKPOINT, V, n, d, j) aj )i (j =£ i) 
Eff: if viewi > v A in-w(n, i) then 

IWi := (ft; U {(CHECKPOINT, u, n, d, j) CT . } 

receive( (view-change, V, n, S, C, P, j) aj )i (j ^ i) 
Eff: let m = (VIEW-CHANGE, v, n, s, C, P, j> CTj . 

if v > viewi A correct-view-change(m, v , j) then 
(«; := im U {m} 



receive( (new- VIEW, V, V, O, N) aj )i (j ^ i) 
Eff: let m = (new- view, v, V, O, N) <Tj 

if v > A v > viewi A correct-new-view(m, v) A -i/ias-new-v/ew^, i) then 
View; := v 
ouu := {} 

(ft; := imUOU NU {m} 
for all (PRE-PREPARE, v, n',m') <Tj 6 (O U N) do 
owfi := o«?i U {(PREPARE, v, n', D{m'),i) <Ti } 
if n' > stable-rti then 
iWi := i«j U {(PREPARE, v, n', D(m'),i) cri } 
update-state-nv(i, v, V, m) 
im := im ^{(request, o,t,c) CTc e im\t < last-rep-t { (c)} 

Internal Transitions 



SEND-PRE-PREPARE(m, V, n)i 
Pre: primaryii) = i A seqno i =n#lA in-iw(t), n, «) A /!fl5-wew-V!ew(u, «)A 

3o, t,c : im= (REQUEST, o, i, c) ar A m 6 in;)A 3(PRE-PREPARE, u, n' ,m) C r i G ;>2 8 
Eff: seqno i := seqno i + 1 

let p = (PRE-PREPARE, v, n, m) <Ti 
outi := o«?i U {p} 
im := im U {p} 



25 



EXECUTE(m, V,n)i 
Pre: n = last-execi + 1 A committed(m, v, n, i) 
Eff: last-exec i := n 
if (m ^ null) then 
let (REQUEST, o, t, c) CTc = m 
if t > last-rep-t i (c) then 
if t > last-rep-t^c) then 
last-rep-t^c) := t 
(last-rep i (c), vah) := g(c, o, va/i) 
owfi := o«?i U {(REPLY, v/ewi, t, c, i, last-rep i (c)) ai } 
m; := irii -&{m} 
if take-chkpt(n) then 
let m' = (CHECKPOINT, vie\Vi,n, D((vah, last-rep i ,last-rep-t i )),i) ai 
outi := <?M?i U {m 1 } 
irii := !«i U {m 1 } 
chkpts i := chkpts i U {(n, (va/i, last-rep^ last-rep-t^)} 

SEND-VIEW-CHANGE(u) i 
Pre: i> = viewi + 1 
Eff: view; := v 

let P' = {(m,v,n)\last-prepared(m,v,n,i)}, 
P = U <m ,„,„) 6 P- (iP = (PREPARE, v, n, D(m), k) ah \p 6 im} U { (pre-PREPARE, v, n, m) apritmryM }), 
C = {m! = (CHECKPOINT, v" ,stable-ni,D(stable-chkpt i ),k) <Tk \m' € irii}, 
m = (VIEW-CHANGE, v, stable-m, stable-chkpt { , C, P, i) ai 
outi := outi U {m} 
irii '■= irii U {m} 

SEND-NEW- VIEW(u, V)i 
Pre: primaryiv) = i A v > viewi A v > A V C im A \V\ = 2/ + 1 A -ihas -new -view (v, i)/\ 
3R : (\R\ = 2/ + 1 A V& 6 # : (3n, s,C,P : (( VIEW-CHANGE, v, n, s, C, P, k)„ h G V))) 
Eff: v/ewi := v 

let O = { (PRE-PREPARE, v, n, m) ai \n > max-n(V) A 3v' : last-prepared(m, v' , n, merge-P(V))}, 
N = {(PRE-PREPARE, v, n, null, k) ai \max-n(V) < n < max-n(0)A 

fa' ,m, n : last-preparedim, v' , n, merge-P(V))}, 
m = (new- view, v, V, O, N) <Ti 
seqno i := max-n(0) 
int := WjUOUJVU {m} 
outi '■= {m} 

update-state-nv(i, v, V, in) 
ini := im <»{ (REQUEST, o, t,c) ac 6 im\t < last-rep-t { (c)} 

COLLECT-GARBAGE; 
Pre: 3R, n,d : (\R\ > f Ai € R AVk € R : (3v : ((CHECKPOINT, v, n, d, k) ah im) 
im <&{m = (PRE-PREPARE, v' ,n' ,m') crj \m 6 im An' < n} 
im <&{m = (PREPARE, v' , n' , d' , j) CT . | m 6 im A n' < n} 
im o{m = (COMMIT, v',n', d', j) aj \m 6 im An' < n} 
im &{m = (CHECKPOINT, v',n',d' ,j) aj \m 6 im An' <n) 
chkpts i := chkpts i ^{p = (n', s)\p £ chkpts i An' < n}) 

5.2 Safety Proof 

This section proves that A gc implements S. We start by introducing some definitions and proving 
an invariant. 
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Eff: int 

im 
in, 
in, 



Definition 5.1 We define the following functions inductively: 

Let 1ZM = {(REQUEST, o,t,c) ac oeOAieNAceCJU {null}, 

r-val : TIM* -+ V 

r-last-rep : TIM* -+ ( C -+ £>') 

r-last-rep-t : TZM* -+ (C->N) 

r-vaZ(A) = u 

Vc G C : (r-last-rep(X)(c) = null-rep) 

V c G C : (r-last-rep-t(\)(c) = 0) 

VyU G 71.M+, 
r-val(p.null) = r-val(p) 
r-last-rep(p.null) = r-last-rep(p) 
r-last-rep-t(p.null) = r-last-rep-t(p) 

V (request, o,i,c) (7c G TZM, p G 7\LM + , 

Vc' ^ c : (r-tof-rep(/u. (REQUEST, o, t,c) (Tc )(c') = r-last-rep(p)(c')) 
Vc' ^ c : (r-tof-rep-^/u. (REQUEST, o, t,c) (Tc )(c') = r-last-rep-t(p)(c')) 
if t > r-last-rep-t(p)(c) then 
let (r, s) = g(c,o,r-val(p)) 
r-val(p. (REQUEST, o, t,c)^ c ) = s 
r-last-rep(p. (REQUEST, o, t,c) ac )(c) = r 
r-last-rep-t(p. (REQUEST, o, t,c) (Tc )(c) = t 
else 
r-va/(/u. (REQUEST, o, i,c) CT< .) = r-val(p) 
r-last-rep(p.(REQUEST,o,t,c) a<1 )(c) = r-last-rep(p)(c) 
r-last-rep-t(p. (REQUEST, o, t,c) (T< .)(c) = r-last-rep-t(p)(c) 

Definition 5.2 We define the following subsets of Ai and predicate: 

Wire = {m \ 3 X : {(m,X) G wire) } 
Wire+o = Wire U { m | 3 j G 7?. : (-^faulty. A m G OKf? ) } 
Wire+io = Wire+o U { m | 3 j G 1Z : (-faulty. A m G (»?) } 
committed- Wire ( s, l,t,n, v,p) = 
3m\...m n = p G 7£.A/f* : (s = r-val(/x) A / = r-last-rep(/z) A t = r-last-rep-t (p) A 
\/0 < k < n : (3v' < v, R : (\R\ > 2/ A 

y q E R : ((COMMIT, v' , k, D(m k ),q) aq G Wire+o)) 
A(3v' < v : ((PRE-PREPARE,t; / ,fc,m A: ) (Tprimary(i) , ) G Wire+o) 
V m k G Wire+o))) 

Invariant 5.3 The following is true of any reachable state in an execution of A gc : 

l.Vi 6 7?. : ((-^faulty t A n-faulty < f) => 

3 /z G 7?..M* : committed-Wire^alijlast-rep^last-rep-t^last-exeCijViewi,/!)) 

2.Vi e 1Z : (-'faulty t A n-faulty < /) => 

V (CHECKPOINT, v,n,D((s,l,t)),i) ai G iV : (3 /z G 7\LM* : committed-Wire(s,l,t,n,v,p)) 
where: 
N = {m | me W;>e+/o V 3 (VIEW-CHANGE, v, n,s,C, P,j)a d G Wire+io: (m G C) V 

3 (new- VIEW, ^,7, 0,N) (Tj G Wire+io : (3 (VIEW-CHANGE, v,n,s,C, P, q) (7q G V : (m G C)) }, 
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Proof: The proof is by induction on the length of the execution. For the base case, the initializations 
ensure that va/j = r-val{\), last-rep i = r-last-rep(X), and last-rep-t i = r-last-rep-t{\). Therefore, 
1 is obviously true in the base case and 2 is also true because all the checkpoint messages 
(checkpoint, v, n, D((s, /, t)),i) ai 6 N have s = vak, I = last-rep t ,t = last-rep-t t . 

For the inductive step, assume that the invariant holds for every state of any execution a of 
length at most /. We will show that the lemma also holds for any one step extension a\ of a. 
The only actions that can violate 1 are actions that change vak, last-rep i , last-rep-t^last-execi, 
decrement viewi, or remove messages from Wire+o. But no actions ever decrement viewi. 
Similarly, no actions ever remove messages from Wire+o because wire remembers all messages 
that were ever sent over the multicast channel and messages are only removed from out ,■ (for any 
non-faulty replica j) when they are sent over the multicast channel. Therefore, the only actions 
that can violate 1 are: 

1. receive( (new- VIEW, V, V, O, N) aj ); 

2. EXECUTE(m, V,n)i 

3. SEND-NEW-VIEW(«, V), 

The inductive hypothesis of condition 2 ensures that actions of the first and third type do not 
violate condition 1 because they set vak , last-rep i , last-rep-t i and last-execi to the corresponding 
values in a checkpoint message from a non-faulty replica. 

Actions of the second type also do not violate 1 because of the inductive hypothesis, 
and because the executed request, m n , verifies committed(m n ,v,n,i) for v < viewi and 
n = last-execi + 1. Since committed(m n ,v,n,i) is true, the 2/ + 1 commits and the pre- 
prepare (or m n ) necessary for committed-Wire to hold are in im. These messages were either 
received by i over the multicast channel or they are messages from i, in which case they are in 
outi or have already been sent over the multicast channel. 

The only actions that can violate condition 2 are those that insert checkpoint messages in N: 

1. RECEIVE( (CHECKPOINT, V, n, d, i) tTi )j 

2. RECEIVE( (VIEW-CHANGE, V, n, S, C, P, q)a q )j 

3. receive( (new- view, v, V, O, N) <Tq )j 

4. SEND(m, R)i 

5. EXBCUTE(m,v,n)j 

6. SEND-VIEW-CHANGE(v)j 

7. SEND-NEW-VIEW(t), V)j 

where j is any non-faulty replica. Actions of types 1, 2, 4, and 6 preserve 2 because the 
checkpoints they insert into N are already in N before the action executes and because of the 
inductive hypothesis. Actions of types 3 and 7 may insert a new checkpoint message from j 
into N; but they also preserve condition 2 because this message has the same sequence number 
and checkpoint digest as some checkpoint message from a non-faulty replica that is already in N 
before the action executes and because of the inductive hypothesis. Finally, the argument to show 
that actions of the fifth type preserve 1 also shows that they preserve condition 2. □ 



Invariant 5.4 The following is true of any reachable state in an execution of A: 
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n-faulty < f => V /i, /i' € H.M* : ((3 s, l,t,v,s',l',t',v' : (committed-Wire(s,l,t,n,v, fi) A 
committed-Wire(s' ,V ,t' ,n' ,v' , //)) A /j,. length < /j, 1 .length) =>• 3//' € 7?..M* : (// = H-n")) 



Proof: (By contradiction) Suppose that the invariant is false. Then, there may exist some sequence 
number k (0 < k < p,. length) and two different requests m*., and m^ such that: 

3v u Ri : (\Ri\ > 2f AVq e Rt : ({COMMIT, v u k,D(m kl ),q) (Tq e Wire+o)) and 
3w 2 , R 2 : (I-R2I > 2/ A V<? G i? 2 : ((commit,^, k,D(m k2 ), q) (Tq e Wre+o)) 

This, Invariant 4.1 and Invariant 4.6 contradict Invariant 4.10. D 



Theorem 5.5 A gc implements S 

Proof: We prove that A gc implements A, which implies that it implements S (Theorems 4.19 
and 4. 14.) The proof uses a forward simulation H from A' to A' (A' is equal to A gc but with all 
output actions not in the external signature of S hidden.) 

Definition 5.6 H is a subset o/states(^4' ) x states(^4'); (x, y) is an element ofH if and only if 
all the following conditions are satisfiedfor any replica i such that a;. faulty j = false, and for any 
replica j: 

1. The values of the state variables in y are equal to the corresponding values in x except for y.wire, y.irii and 
y.outi. 

2. y.irii <& {m = (PRE-PREPARE, v, n, m) l7j V m = (PREPARE, v,n,d,j) aj V 

m = (COMMIT, v,n,d,j)cr \m £ y.im A n < x.stable-m} 
«{m|m 6 y.irii A (tag(m, VIEW-CHANGE) V tag(m, NEW-VIEW))} 
= x.itii «■ {m = (PRE-PREPARE, v, n, m) (7j V m = (PREPARE, v,n,d,j) aj V 
m = (cOMM\T,v,n,d,j) a . \m £ x.im A n < x.stable-rii} 
#{m|m 6 x.im A {tag(m, CHECKPOINT) V tag(m, VIEW-CHANGE) V tag(m, NEW- VIEW))} 

3. Let consistent-vc(m l ,rn?) = 

3v,n,s,l,t,C,P,P',j : (m [ = (VIEW-CHANGE, v,n, {s, l,t), C, P, j) crj A 
m 2 = {v\EW-CHANGE,V,P',j) crj A 

A' gc .correct-view-change(m' ,v, j) <$=> (A' .correct-view-change(m 2 , v, j) A 

P = P' <&{m = (PRE-PREPARE, v' ,ri ,m') ak Vm = (PREPARE, v',n',d', k) ak \ m £ P' An' < n}))) 
consistent-vc-set(M i , M 2 ) = 

Vm 1 6 M 1 : (3 m 2 £ M 2 : consistent-vc(m\m 2 )) A 
Vm 2 6 M 2 : (3 m 1 6 M 1 : consistent-vc(m l ,m 2 )), 
andlety.vci = { (VIEW-CHANGE, v, P, j) aj £ y.irii}, 

X.Vd = {{VlEW-CHANGE,V,n, {s, I, t),C,P,j) c , j £ X.im} 
then consistent-vc-set(x.vCi,y.vCi) is true 

4. Let consistent-nv-set(M\ , M2) = 

M 2 = {m 2 = (new-view, v,V',0',N') aj I 

3 m 1 = (NEW-VIEW, v, V, O, N) aj £ Mi : (consistent-vc-set(V,V) A 
A' gc .correct-new-view(m l , v) -4=> (A' .correct-new-view(m 2 ,v) A 

O = O' ■&■ {m = (PRE-PREPARE, v,n, m') <Tj \ m £ O' A n < max-n(V)} A 
N = N' ■» {m = (PRE-PREPARE, u, n, m') <Tj \ m £ N' A n < max-n(V)}))} , 
andlety.wvi = {(NEW- VIEW, v, V, O, N) aj £ y.im}, 

x.nvi = {(new- view, v,V, O, N) aj £ x.im} 
then consistent-nv-set(x.nvi,y.nvi) is true. 
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5. Let consistent-all{M\ M 2 ) = 

Vm € M 1 : (3 m' 6 M 2 : {tag(m, VIEW-CHANGE) A consistent-vc(m,m'))V 

(tag(m, NEW- VIEW) A conitoenf-nv-ie?({m}, {m'})) V 

(-itag(m, VIEW-CHANGE) A ^tag(m, NEW- VIEW) A m = m')), 
Xi = ai.owfi U {(m)^; | (m),^ 6 x.Wire} &{m \ tag(m, CHECKPOINT)}, 
and Yi = y.outi U {{m) <Ti \ {m) <Ti 6 y.Wire}, 
then consistent-all(Xi. Yi) 

6. LetXf au ify = { (m) <Tj \ x. faulty ^ A (m) CTj € x.Wire}, 

Y faulty = i ( m )"i I y-/ "'^ A (m) <Tj 6 t/.Wre}, 
consistent-all{Xj aulty , Yf auIty ) 

7. V (r) CTc G x.Wire : (3 <r) CTc € y.Wire) 

Additionally, we assume faulty automata in x are also faulty and identical in 7i[x] (i.e., they 
have the same actions and the same state.) Note that the conditions in the definition ofH only 
need to hold when n-faulty < /, for n-faulty > / the behavior of S is unspecified. 

To prove that fi is in fact a forward simulation from A' to A' one most prove that both of 
the following are true: 

1. For all x G start(A' gc ), W[x] n start(A') ^ {} 

2. For all (x, n, x') E trans(A ' ), where a; is a reachable state of A', and for all y E H[x\, 
where y is reachable in A', there exists an execution fragment a of A' starting with y and 
ending with some y' E 7i[x'] such that trace(a) = trace(n). 

Condition 1 holds because (x, y) E H for any initial state x of A' and y of A'. It is clear 
that x and y satisfy the first clause in the definition of H because the initial value of the variables 
mentioned in this clause is the same in A' and A'. Clauses 2 to 7 are satisfied because x.im only 
contains checkpoint messages, and y.irii, x.outi, y.outi, x.wire, and y.wire are empty 

We prove condition 2 by showing it holds for every action of A' . We start by defining 
an auxiliary function (3(y, m, a) to compute a sequence of actions of A' starting from state y to 
simulate a receive of message m by an automaton a (where a is either a client or replica identifier): 

f3(y,m,a) = 
if3X :((m,X) 6 y.wire) then 
if3X : ((m, X) 6 y.wire A a e X) then 

RECEIVE(m) a 
else 
MISBEHAVE(m, X, X U {a}). RECEIVE(m) a | (m, X) e y.wire 
else 
if 3 i : (y. faulty i = false A m € y.outi) then 

SEND(m, {a})i. RECEIVE(m) a 
else 
_L 

If RECElVE(m) a is enabled in a state x, there is an m' such that (3(y, m', a) is defined and the 
actions in (3(y, m', a) are enabled for all y E H[x], and: 
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• m = m', if m is not a checkpoint, view-change, or new-view message 

• consistent-vc(m, m'), if m is a view-change message 

• consistent-nv-set({m}, {m 1 }), if m is a new-view message 

This is guaranteed by clauses 5, 6, and 7 in the definition of 7i. 

Now, we proceed by cases proving condition 2 holds for each n G acts(A' ) 

Non-faulty proxy actions. If n is an action of a non-faulty proxy automaton P c other than 
RECElVE(m = (reply, v,t,c, i, r) (Ti ) c , let a consist of a single it step. For the receive actions, let 
a = (3(y, m, c). In either case, when it is enabled in x all the actions in a are also enabled starting 
from y and an inspection of the code shows that the state relation defined by H is preserved in all 
these cases. 

Internal channel actions. If 7r is a MiSBEHAVE(m, X, X') action, there are two cases: if ir is 
not enabled in y, let a be A; otherwise, let a contain a single n step. In either case, H is preserved, 
because n does not add new messages to x.Wire. 

Receive of request, pre-prepare, prepare, or commit. For actions ir = RECEiVE(m)j where 
m is a syntactically valid request, pre-prepare, prepare, or commit message, let a = (3(y, m,i); a 
transforms y into y' G H. \x'\ : 

• 7r and a modify wire in a way that preserves clauses 5, 6, and 7. 

• For receives of request messages, a and ir add the same messages to outi and im thereby 
preserving the state correspondence defined by 7i. 

• For the other message types, the definition of 7i and the definition of in-wv ensure that when 
the first if condition is true in x, it is also true in y (because the condition is more restrictive 
in A' and x.itii and y.im have the same prepare and commit messages with sequence 
numbers higher than x.stable-ni.) Thus, in this case, the state correspondence defined by 
fi is preserved. But it is possible for the if condition to be true in y and false in x; this 
will cause a message to be added to y.irii and (possibly) y.outi that is not added to x.ini or 
x.outi. Since this happens only if the sequence number of the message received is lower 
than or equal to x.stable-ni, the state correspondence is also preserved in this case. 

Garbage collection. If n = receive( (checkpoint, v,n,d, j) a )«, or ir = collect- 
GARBAGEj, the condition holds when a is A. It is clear that the condition holds for the first 
type of action. For the second type, the condition is satisfied because all the messages removed 
from x.irii have sequence number lower than or equal to n and the action sets x.stable-ni to n. 
The action sets x.stable-ni to n because it removes all triples with sequence number lower than n 
from x.chkpts i and there is a triple with sequence number n in x.chkpts^ The existence of this 
triple is guaranteed because the precondition for the collect-garbage^ action requires that there 
is a checkpoint message from i with sequence number n in x.irii and i only inserts checkpoint 
messages in in% when it inserts a corresponding checkpoint in chkpts^ 

Receive view-change. If ir = RECEiVE(m = (view-change, v,n,s, C,P,j) a )i, let a = 
(3(y, m' , i) such that consistent-vc(m, m'). The definition of consistent-vc ensures that either both 
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messages are incorrect or both are correct. In the first case, n and a only modify the destination 
set of the messages in wire; otherwise, they both insert the view change message in im. In either 
case, the state correspondence defined by ~H is preserved. 

Receive new-view. When n = RECEiVE(m = (new- VIEW, v, V, 0,N) a .)i, we consider 
two cases. Firstly, if the condition in the outer if is not satisfied, let a = (3(y,m',i), where 
consistent-nv-set({m} , {m'}). It is clear that this ensures y' G H[x'] under the assumption that 
y ^H[x\. Secondly, if the condition in the outer if is satisfied when n executes in x, let a be the 
execution of the following sequence of actions of A': 

1. The actions in /3(y, m! = (NEW- VIEW, v, V' , O 1 ,N') <Tj , i), where consistent-nv-set({m} , {m'}) 

2. Let C be a sequence of tuples (v„ , R„ , m„ ) from Nx2 K x TZM such that the following conditions are true: 
i) Vn : (x.last-execi < n < max-n(V)) 

ii)V (v n ,R n ,m n ) : (v n < v A \R„\ > 2/ A V k 6 R n : ((COMMIT, v n , n, D(m n ), k) ah 6 x.Wire+o) 

A(3w' : ((PRE-PREPARE, v' ,n, m n )„ im ary(vl) 6 x.Wire+o) V m„ 6 x.Wire+o) 
for each (v„ , R„ , m„ ) € C in order of increasing n execute: 

a)f3(y,c nk = (COMMIT, v n ,n,D(m n ),k) ak ,i), for each k 6 R n 

b) if enabled (3(y,p n = (PRE-PREPARE, v',n, m n ) a p rimar y ( ^, ) ,i) else (3(y,m n ,i) 

c) EXECUTE(m„,u„,n)j 

The definition of 71 (clauses 1,4,5 and 6) ensures that, when the receive of the new-view 
message executes in y , the condition in the outer if is true exactly when it is satisfied in x. Let y\ be 
the state after (3(y, m', i) executes; we show that when C is empty (i.e., max-n(V) < last-execi), 
y' = y\ G 'H[x']. This is true because: 

• Both 7r and P(y,m',i) set viewi to v, add all the pre-prepares in O U N to im, and add 
consistent new-view messages to im. 

• (3(y, m' , i) also adds the pre-prepares in (O 1 U N') <=^(0 U N) to im but this does not violate 
H. because n ensures that x 1 .stable-m is greater than or equal to the sequence numbers in 
these pre-prepares. 

• Both 7T and (3(y, vn! , i) add prepares to im and outf, (3(y, m', i) adds all the prepares added 
by 7r and some extra prepares whose sequence numbers are less than or equal to x' .stable-m. 

When C is not empty (i.e., max-n(V) > last-execi), it is possible that y\ ^ 7i[x'] because 
some of the requests whose execution is reflected in the last checkpoint in x' may not have executed 
in J/1. The extra actions in a ensure that y' 6 'H[x']. 

We will first show that C is well-defined, i.e., there exists a sequence with one tu- 
ple for each n between x.last-execi and max-n(V) that satisfies conditions i) and ii). Let 
m" = {view-CHANGE, v, max-n(V), (s, I, t), C", P, k) ak be the view-change message in V whose 
checkpoint value, (s,l,t), is assigned to (valijast-rep^last-rep-t^. Since m" is correct, C" 
contains at least / + 1 checkpoint messages with sequence number max-n(V) and the digest of 



32 



(s, I, t). Therefore, the bound on the number of faulty replicas, and Invariant 5.3 (condition 2) 
imply there is a sequence of requests /^i such that committed-Wire(s, I, t, max-n(V),v,p,i). 

Since by the inductive hypothesis y G fi[x], all the the commit, pre-prepare and request 
messages corresponding to p,\ are also in y.Wire+o. Therefore, all the actions in a) and at least 
one of the actions in b) are enabled starting from y\ for each n and each k G R n . Since v n < v 
for all the tuples in C, each receive in /3(y, c nk , i) will insert c nk in im. Similarly, the receive 
of the pre-prepare or request will insert a matching pre-prepare or request in im. This enables 
execute (m n , v n ,n)i. 

Invariant 5.3 (condition 1) also asserts that there exists a sequence of requests p,2 such 
that committed-Wire(x.vali,x.last-rep i ,x.last-rep-t i ,x.last-execi,x.viewi,H2)- Since by the inductive 
hypothesis y G fi[x], all the the commit, pre-prepare and request messages corresponding to i*i 
and H2 are also in y.Wire+o. This and Invariant 5.4 imply that \i2 is a prefix of fj,\. Therefore, 
after the execution of a, vah , last-rep $ , last-rep-ti , last-execi have the same value in x' and y' as 
required by %. 

Send. If 7r = SEND(m, X)u let a be: 

• A single send(m, X)i step, if m does not have the CHECKPOINT, VIEW-CHANGE, or NEW- VIEW 
tag and this action is enabled in y. 

• A, if m has the CHECKPOINT tag or the action is not enabled in y (because the message is 
already in the channel.) 

• A single send(m', X)i step, if m has the VIEW-CHANGE tag and this action is enabled in y 
(where consistent-vc(m, m 1 ).) 

• A single send(m', X)i step, if m has the NEW- VIEW tag and this action is enabled in y (where 
consistent-nv-set({m} , {m'}).) 

Send-pre-prepare and send-commit. If n = SEND-PRE-PREPARE(m, v, n)i or ir = send- 
COMMIT(m, v, n)i, let a contain a single n step. This ensures y' G 7i[x'] because these actions 
are only enabled in x when they are enabled in y, and they insert and remove the same messages 
from im and outi. 

Execute. When n = EXECUTE(m, v, n)i, let a contain a single n step. The action is enabled 
in y when it is enabled in x because it is only enabled in x for n > x.stable-ni and x.im and y.im 
have the same pre-prepare and commit messages with sequence numbers greater than x.stable-ni 
and the same requests. It is easy to see that the state correspondence defined by % is preserved by 
inspecting the code. 

View-change. If n = vlEW-CHANGE(t>)j, let a contain a single n step. The action is enabled 
in y when it is enabled in x because viewi has the same value in x and y. Both 7r and a insert view- 
change messages m and m' (respectively) in im and outi ; it is clear that this ensures y' G % [x 1 ] 
provided consistent-vc(m' , m') is true. Clause 2 in the definition of H ensures that m and m' 
contain the same messages in the P component for sequence numbers greater than x.stable-nf, 
therefore, consistent-vc(m' , m') is true. 

Send-new-view. If ir = SEND-NEW- vlEw(t>, V)i, let a be the execution of the following 
sequence of actions of A 1 : 
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1. send-new-view(i>, V')i step, where consistent-vc-set(V, V). 

2. Let G be a sequence of tuples (v„ , R„ , m„ ) from Nx2 x x 1ZM such that the following conditions are true: 
i) Vn : (x.last-execi < n < max-n(V)) 



ii)V (v n ,R n ,m n ) : (v n < v A \R„\ > 2/ A V k e R n : ((COMMIT, v n , n, D(m n ), k) ak 6 x.Wire+o) 

A (3 v' : ((PRE-PREPARE, v' ,n, m„) a - m ary(vl) € x.Wire+o) V m„ 6 ai.Wi're+o) 
for each (u n , i? n , m„ ) € C in order of increasing n execute: 

a)/3(y,c nk = (COMMIT, v n ,n,D(m n ),k) ak ,i), for each k G R„ 

b) if enabled (3(y,p n = (PRE-PREPARE, v',n, m n ) a ima vt) , i) else /3(y, m n , «) 

c) EXECUTE(m n , v„,n). 

This simulation and the argument why it preserves H is very similar to the one presented for 
receives of new-view messages. 

Failure. If n = REPLICA-FAILUREj or ir = CLIENT-FAILURE i, let a contain a single n step. It 
is easy to see that y' 6 %[§']. 

Actions by faulty nodes. If n is an action of a faulty automaton, let a contain a single n step. 
The definition of H. ensures that a is enabled in y whenever 7r is enabled in x. Modifications to 
the internal state of the faulty automaton cannot violate H.. The only actions that could potentially 
violate H are sends. But this is not possible because a faulty automaton cannot forge the signature 
of a non-faulty one. Q 
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